Add CHASM Visibility registration and task processing#8491
Add CHASM Visibility registration and task processing#8491awln-temporal merged 48 commits intomainfrom
Conversation
4a8af63 to
c07d41e
Compare
bergundy
left a comment
There was a problem hiding this comment.
Started reviewing but I think we still don't agree on the API, so let's discuss that before and agree before implementing anything.
560562e to
c103f5f
Compare
bergundy
left a comment
There was a problem hiding this comment.
This is already very close.
Please document all exported fields from the chasm package.
bergundy
left a comment
There was a problem hiding this comment.
Approved with comments. Overall this looks great. Please add the missing docstrings and test coverage before merging.
|
|
||
| var ( | ||
| PayloadTotalCountSearchAttribute = chasm.NewSearchAttributeInt(PayloadTotalCountSAAlias, chasm.SearchAttributeFieldInt01) | ||
| PayloadTotalSizeSearchAttribute = chasm.NewSearchAttributeInt(PayloadTotalSizeSAAlias, chasm.SearchAttributeFieldInt02) |
There was a problem hiding this comment.
Can you use one attribute that is an alias to an existing pre-defined field for completeness? I feel like you're not understanding what I'm asking.
There was a problem hiding this comment.
apologies, there was a reference to use a "predefined" field (not TemporalInt01 etc) in test_component_test.go. I also added a reference to the predefined field SearchAttributeTemporalScheduledByID.
As for whether to include it in Payload.go, there was a previous TODO to track Count and Size as search attributes, and I wasn't sure that adding a reference to TemporalScheduledById made sense within the scope of the Payload component. This component only tracks the current count and size of payloads, and a map from the PayloadKey to its Payload blob. What would be the values provided to the search attribute?
I agree that functional tests should cover this case, I just wasn't sure if it was acceptable style to add a search attribute that doesn't seem related to the component. Added a reference in payload.go to this.
| // In SQLite, keyword list can return a string when there's only one element. | ||
| // This changes it into a slice. |
There was a problem hiding this comment.
I suppose you can't just skip the CHASM search attributes based on this comment.
There was a problem hiding this comment.
This is temporary, we won't skip CHASM search attributes once we consolidate the type maps on the query path.
| saType, err = typeMap.getType(saName, customCategory|predefinedCategory) | ||
| if err != nil { | ||
| if err != nil && !IsChasmSearchAttribute(saName) { | ||
| lastErr = err | ||
| continue | ||
| } |
There was a problem hiding this comment.
Ditto, I think it's better that you check if it's a CHASM search attribute before checking the type. Also, in this code, if it's a CHASM search attribute, saType will be UNSPECIFIED, so you'd be setting the wrong type for CHASM search attributes.
There was a problem hiding this comment.
If UNSPECIFIED, SetMetadataType will return early. To be clear, this is just temp code that will need to change when the query path is implemented.
func SetMetadataType(p *commonpb.Payload, t enumspb.IndexedValueType) {
if t == enumspb.INDEXED_VALUE_TYPE_UNSPECIFIED {
return
}
_, isValidT := enumspb.IndexedValueType_name[int32(t)]
if !isValidT {
panic(fmt.Sprintf("unknown index value type %v", t))
}
p.Metadata[MetadataType] = []byte(enumspb.IndexedValueType(t).String())
}
There was a problem hiding this comment.
This is in the write path, not query path. What I mean is this code will always set the type for CHASM search attribute as UNSPECIFIED, and as you pointed out, the function will just ignore, ie., the CHASM search attributes will never have its type set in the metadata.
Or did I miss something?
There was a problem hiding this comment.
Encode is on the read path. I suppose either way, the MetadataType is somewhat ignored on the write path, and we basically reconstruct the MetadataType solely from the saTypeMap right now? Either way, without merging the saTypeMap with the CHASM component's saTypeMap, current functionality won't work. On the next PR to support querying, we can re-evaluate how we want to support setting the MetadataType. This portion of code is mainly to get functional tests to succeed.
| // If the search attribute name is not in typeMap but the payload has type metadata, | ||
| // we can still decode it (e.g., for CHASM search attributes). | ||
| if _, hasTypeMetadata := saPayload.Metadata[MetadataType]; !hasTypeMetadata { | ||
| lastErr = err | ||
| } |
There was a problem hiding this comment.
ditto
| // If the search attribute name is not in typeMap but the payload has type metadata, | |
| // we can still decode it (e.g., for CHASM search attributes). | |
| if _, hasTypeMetadata := saPayload.Metadata[MetadataType]; !hasTypeMetadata { | |
| lastErr = err | |
| } | |
| // If the search attribute name is not in typeMap but the payload has type metadata, | |
| // we can still decode it (e.g., for CHASM search attributes). | |
| saType = saPayload.Metadata[MetadataType] | |
| if saType == enumspb.INDEXED_VALUE_TYPE_UNSPECIFIED { | |
| lastErr = err | |
| } |
There was a problem hiding this comment.
this isn't necessary, DecodeValue handles this case
func DecodeValue(
value *commonpb.Payload,
t enumspb.IndexedValueType,
allowList bool,
) (any, error) {
if t == enumspb.INDEXED_VALUE_TYPE_UNSPECIFIED {
var err error
t, err = enumspb.IndexedValueTypeFromString(string(value.Metadata[MetadataType]))
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidType, t)
}
}
switch t {
case enumspb.INDEXED_VALUE_TYPE_BOOL:
return decodeValueTyped[bool](value, allowList)
case enumspb.INDEXED_VALUE_TYPE_DATETIME:
return decodeValueTyped[time.Time](value, allowList)
case enumspb.INDEXED_VALUE_TYPE_DOUBLE:
return decodeValueTyped[float64](value, allowList)
case enumspb.INDEXED_VALUE_TYPE_INT:
return decodeValueTyped[int64](value, allowList)
case enumspb.INDEXED_VALUE_TYPE_KEYWORD:
return validateStrings(decodeValueTyped[string](value, allowList))
case enumspb.INDEXED_VALUE_TYPE_TEXT:
return validateStrings(decodeValueTyped[string](value, allowList))
case enumspb.INDEXED_VALUE_TYPE_KEYWORD_LIST:
return validateStrings(decodeValueTyped[[]string](value, false))
default:
return nil, fmt.Errorf("%w: %v", ErrInvalidType, t)
}
}
There was a problem hiding this comment.
We need to revert this change. This is swallowing errors when the custom search attribute is not defined in the typeMap.
The correct behavior is: if the typeMap input is not nil, Decode checks everything against typeMap; otherwise, it uses the payload metadata if available.
There was a problem hiding this comment.
just fmi, in which cases would the typeMap be nil, and we would then be permitted to decode the value based off the payload Metadata? I would have assumed we would've thrown an error when translating from alias to field upstream, but it seems that isn't the case.
| func Decode( | ||
| searchAttributes *commonpb.SearchAttributes, | ||
| typeMap *NameTypeMap, | ||
| allowList bool, | ||
| ) (map[string]interface{}, error) { |
There was a problem hiding this comment.
Please add unit tests for changes you did.
There was a problem hiding this comment.
there are unit test validations for the changes. Previously, key2 would have failed decoding due it missing in the saTypeMap
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
## What changed? Fix the search attributes `Decode` function to not swallow errors when the type map is passed to the function. ## Why? Recent PR #8491 changed the behavior allowing search attributes with metadata to passthrough when the type map is not nil and the custom search attribute is not defined. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks
What changed?
Adds CHASM search attribute provider and mapper implementation. To use CHASM search attributes, embed your component with ComponentSearchAttributesProvider and register the search attributes to the CHASM Registry.
To define a component search attribute, define a SearchAttribute as a package/global scoped variable (CHASM search attributes should reference exported variables like
SearchAttributeFieldInt01,SearchAttributeFieldDateTime01:To update SearchAttributes during an execution of a CHASM archetype, implement the provider
SearchAttributesin the root component, and call theNewValuemethod on a SearchAttribute within the Provider method to generate a new SearchAttributeValue. On SearchAttribute updates during CHASM transactions, the framework will detect changes and submit a Visibility task for persistence.To register the search attribute with the CHASM Registry,
WithSearchAttributesmust be passed as a RegistrableComponentOption to the root component of the library. This is required to support Visibility queries.Why?
Required to support CHASM Search Attributes.
How did you test it?
Potential risks
TBD